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SUMMARY 



The purpose of this grant was to expand upon an already-developed 
process for teacher-course evaluation. The process uses student 
ratings collected with an instrument which focuses on teacher be- 
haviors and activities, overall assessments, and student and course in- 
formation. The grant supported the development/acquisition of teach- 
ing improvement materials (to be used alone or in conjunction with 
the evaluation) and a computer-based mechanism which provides 
statistical results, generates narrative reports interpreting the statis- 
tics, and matches areas of weak performance with the specific refer- 
ences and resources at hand. The system is PC-based and can be 
adapted to the needs and unique requirements of any institution 
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PROJECT PRODMCTS: 



• "Effective Use of Time in the College Classroom" a 26-minute 
videotape with accompanying guide/workbook. 

• "Writing a Syllabus", a monograph/self-instructional unit. 

• "Selecting Texts and Readings", a monograph/self-instructional unit. 

• "Motivating Students", a monograph/self-instructional unit. 

• "Handbook for Teachers" A basic resource on college teaching in 
'template' form (on PC disk in 'Ventura" desktop publishing format) 
ready to be adapted with information provided by any institution and 
used in a teaching improvement program. 

• A partially annotated bibliography of over 1000 teaching-learning 
resources (available on PC disk or in print). 

• A bibliography of over 1000 items on evaluation, student ratings, and 
associated topics (available on PC disk or in print). 

• The Teacher-Course Evaluation Project (TCEP) questionnaire a 
diagnostic instrument which can provide data usable for teaching 
improvement, student course selection, administrative decision making, 
institutional research, or program evaluation. 

• Evaluation system software and utilities which can be used for: 

• evaluations, surveys, or other kinds of data collection; 

• generating reports in many formats rangin^^ from student catalogs 
of evaluation results to department/college comparisons of cour- 
ses, individuals, programs, etc.; 

• standard statistical analysis of student ratings providing distribu- 
tions, means, deviations, and ranges of results 

• special analyses for teaching improvement, institutional research, 
evaluation/teaching research etc.; 

© database management; 

• production of data files for use with SPSS-X statistical packages 

• office procedures such as keeping logs of evaluations, producing 
mailing lists and lables, etc. 

• Research, presentations at professional meetings, materials 



EXECUTIVE SUMMARY 
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comprehensive teaching improvement system 



GRANTEE: The Office of Instructional Development and Evaluation 

417 Dodge Library Northeastern University 
360 Huntington Ave. Boston, MA, 02115 



PROJECT DIR.: Michael Theall, PhD 
617-437-4897 



A. PROJECT OVERVIEW 

This project continued a Northeastern University effort to develop a valid, reliable, accurate, 
and efficient system for evaluating and improving teaching. The system has three basic parts: 
1) procedures, and resources for collecting, managing, analyzing, and reporting data; 2) 
strategies and resources for teaching improvement; and 3) special software which models the 
teaching consultation process to interpret statistical analyses and to provide narrative as well 
as numeric information which can more clearly identify areas of weakness, suggest generally 
successful teaching techniques or strategies, and match specific resources to the instrL::tor's 
needs. Given that the system can provide a 'baseline' teaching improvement service to many 
faculty, then the time of professional evaluation/development staff can be more directed to 
high impact projects such as training teaching assistants or working with departments; to 
more focused activities such as assistance to faculty with special reeds; to institutional 
research; or to further development projects. 

The word "system" is important because it denotes a comprehensive and interrelated set of 
resources, procedures, and products. Just as a "computer system" involves more than 
machines, the "evaluation system" planned for Northeastern involves more than a 
questionnaire and machine readable answer sheets. The FIPSE project was focused on the 
development Of the resources which would make up the development and teaching 
improvement parts of the "system". 



B. PURPOSE 

Although the direct focus of the system is the evaluation and iniprovement of teaching, it can 
be used for other purposes in order to serve many clients. 

It serves faculty through its direct teaching improvement functions, its ability to provide data 
appropriate for inclusion in promotion / tenure / merit dossiers, and its ability to identify 



excellent teachers for awards, master teacher programs, and similar faculty development 
activities. 

It serves students by providing an efficient means of conducting evaluations without having 
to rely on assistance from student organizations and also, with its ability to produce 
publishable reports of course evaluation results for student use in course selection. 

It serves academic administrators through its ability to produce comparative reports of 
individual results using the most appropriate databases (thus providing valid data for 
promotion, tenure, and merit decisions); through its ability to produce reports of overall 
department ^nd/or college performance; and through its ability to produce reports based on 
subgroups of data (such as ratings only from graduate students; ratings only from students 
majoring in a given field; ratings only of teaching assistants). 

It serves the institution through its ability to collect, analyze, and report data for institutional 
research, program evaluation, classroom research, or curriculum decision making. 

Finally, the system can be used as a pure research tool to gather information for studies on 
teaching, learning, evaluation, and many other topics. 



C. BACKGROUND 

More than two decades of research have ^tablished that students are valid and reliable 
raters of the instruction they receive. At Northeastern, the Teacher-Course Evaluation 
Project (TCEP) had successfully developed and tested an evaluation instrument and 
procedures for data collection, analyses, and reporting. The process was accurate and cost 
efficient. The questionnaire had been extensively analyzed and was found to be valid. 
Analysis of evaluation results showed Northeastern students to be reliable raters whose 
opinions were not significantly affected by demographic variables (e.g., sex, class), by 
performance related variables (e.g., grade-point-average), by contextual variables (e.g., class 
size), or by other variables discussed in the evaluation literature. But support was needed to 
develop the rest of the system. The FIPSE proposal was written to seek that support. 



D. PROJECT DESCRIPTION 

This project followed a systematic process of development in order to assemble or produce a 
set of resources which could supplement the existing process for evaluation to create a 
comprehensive system for evaluating and improving teaching.. Beginning with literature 
reviews and needs analyses, it went on to the design and production of products. Expert 
review and field testing followed, with revision and dissemination as the final steps. Most 
grant products reached the stage of review and/or field testing with information about the 
system concept and one product being disseminated to some extent 



10 



E, PROJECT RESULTS 



During the course of the project, an extensive collection of materials was 
obtained/developed. These included articles, books, reports, audio-visua^ items, computer 
software for development and teaching improvement (e.g., "Ventura" aesx^top publishing 
software, the Tencore^ lesson authoring system, and the 'TARTAR" test 
development/scoring/item banking software package). Equally important was the 
development of special system software which al lowed the power and flexibility required to 
serve the various functions listed above. Although the existing evaluation had been initiated 
on a large computer (Northeastern's VAX 11/780 first, then the VAX 8650), efficiency and 
confidentiality demanded a change to a computing environment which was not used by 
others. The Digital Corporation provided a MICRO- VAX II for this purpose. That machine 
was used for almost the entire grant period. An im portant grant-supported improvement 
was the development of software for the personal computing environment. An 
IBM-PC-based version of the evaluation software has been in service and successfully field 
tested at the State University of New York at Buffalo (SUNYB) since Fall, 1987. Tne PC 
verision will be implemented at Northeastern in the coming quarter. 

As the development process progressed, faculty at Northeastern and SUNYB were first 
provided with diagnostic evaluation reports, ihen, at Northeastern, the special report 
capabilities of the system were used to generate reports comparing individual performance 
against selected norms (e.g., a person teaching.a graduate course of 20 students (,ompared 
only to other faculty in the same college who were teaching graduate courses of the same 
size). This was an important additional service because for the first time, faculty and 
administrators had access to data appropriate for use in merit/tenure/promotion decisions. 

Another example of a product which has already been disseminated is a generic handbook 
for teachers. Based on the Northeastern University "Handbook for Teachers", the generic 
handbook can be adapted through desktop publishing to incorporate information specific to 
any institution while retaining all its general information about teaching and related issues. 
SUNYB and Indiana University already have their own handbooks while the University of 
Massachusetts has commissioned its own, and several other institutions have expressed 
interest in obtaining their own, custom versions. 

Finally, several presentations have hci\n made at professional meetings and the grant was the 
basis of an application to the Sears Roebuck Foundation for support to initiate an evaluation 
and teaching improvement ser/ice for teaching assistants and new faculty at Northeastern. 



R SUMMARY and CONCLUSIONS 

Although it is difficult 'o quantify the outcomes of these beginning efforts, all reports from 
participants or reviewers have been favbrablerThe Northeastern system continues to 
efficiently handle large numbers of evaluations (almost 1500 course evaluations in the past 
quarter). This task was completed with the equivalent of only four full-time staff. At 
SUNYB, the same degrees of satisfaction and efficiency have been achieved. 
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A. PROJECT OVERVIEW 



A "System" for Evaluating and Improving Teaching 

This project continued a Northeastern University effort to develop a valid, reliable, 
accurate, and efficient system for evaluating and improving teaching. The system 
has three basic parts: 1) procedures, and resources for collecting, managing, 
analyzing, and reporting data; 2) strategies and resources for teaching improvement; 
and 3) special software which models the teaching consultation process to interpret 
statistical analyses and to provide narrative as well as numeric information which can 
more clearly identify areas of weakness, suggest generally successful teaching 
techniques or strategies, and match specific resources to the instructor's needs. 
Given that the system can provide a 'baseline' teaching improvement service to 
many faculty, then the time of professional evaluation/development staff can be 
more directed to high impact projects such as training teaching assistants or working 
with departments; to more focused activities such as assistance tc faculty with special 
needs; to institutional research; or to further development projects. 

The word "system" is important because it denotes a comprehensive and interrelated 
set of resources, procedures, and products. Just as a "computer system" involves 
more than machines, the "evaluation system" planned for Northeastern involves 
more than a questionnaire and machine readable answer sheets. The FIPSE project 
was focused on the development of the resources which would make up the 
development and teaching improvement parts of the "system". 



Clients Served by the System 

Although the direct focus of the system is the evaluation and improvement of 
teaching, it can be used for other purposes in order to serve many clients. 

It serves faculty through its dir-^rt teaching improvement functions, its ability to 
provide data appropriate for inclusion in promotion / tenure / merit dossiers, and its 
ability to identify excellent teachers for awards, master teacher programs, and 
similar faculty development activities. 

It serves students by providing an efficient means of ronducting evaluations without 
having to rely on assistance from student organizations and also, with its ability to 
produce publishable reports of course evaluation results for student use in course 
selection. 

It serves academic administrators through its ability to produce comparative reports 
of individual results using the most appropriate databases (thus providing valid data 
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for promotion, tenure, and merit decisions); through its abiUty to produce reports of 
overall department and/or coUege performance;"and through its ability to produce 
reports based on subgroups of data (such as ratings only from graduate studems; 
ratings only from students majoring in a given field; ratings only of teaching 
assistants). 

It serves the institution through its ability to collect, analyze, and report data for 
institutional research, program evaluation, classroom research, or curriculum 
decision making. 

Finally, the system can be used as a pure research tool to gather information for 
studies on teaching, learning, evaluation, and many other topics. 



Beginnings 

When the project began, the first part of the system was essentially complete. An 
evaluation instrument and procedures for data collection, analysis, and reporting 
had already been put in place and had been tested in several hundred courses. The 
process was accurate and cost efficient. The questionnaire had been extensively 
analyzed and was found to be valid. Analysis of evaluation results showed 
Northeastern students to be reliable raters whose opinions were not significantly 
affected by demographic variables (e.g., sex, class), by performance related variables 
(e.g., grade-point-average), by contextual variables (e.g., class size), or by other 
variables discussed in the evaluation literature. 



Progress and Products 

During the course of the project, an extensive collection of materials was 
obtained/developed. These included articles, books, reports, audio-visual items, 
computer software for development and teaching improvement (e.g., "Ventura 
Publisher" desktop publishing software, the 'Tencore" lesson authoring system, and 
the "PAR/AR" test development/scoring/item banking software package). Equally 
important was the development of special system software which allowed the power 
and flexibility required to serve the various functions listed above. Although the 
existing evaluation had been initiated on a large computer (Northeastern's VAX 
11/780 first, then the VAX 8650), efficiency and confidentiality demanded a change 
^ to a computing environment which was not used by others. The Digital Corporation 
provided a MICRO- VAX II for this purpose. That machine was used for almost the 
entire grant period. An important grantrsupported improvement was the 
development of software for the personal computing environment. An 




IBM-PC-based version of the evaluation software has been in service and 
successfully field tested at the Stale University of New York at Buffalo (SUNYB) 
since Fall, 1987. The PC version will be installed at Northeastern in the coming 
quarter. 

As the development process progressed, faculty at Northeastern and SUNYB were 
first provided with diagnostic evaluation reports, then, at Northeastern, the special 
report capabilities of the system were used to generate reports comparing individual 
performance against selected norms (e.g., a person teaching a graduate course of 20 
students compared only to other faculty in the same college who were teaching 
graduate courses of the same size). This was an important additional service 
because for the first time, faculty and administrators had access to data appropriate 
for use in merit/tenure/promotion decisions. 

Another example of a product which has already been disseminated is a generic 
handbook for teachers. Based on the Northeastern University "Handbook for 
Teachers", the generic handbook can be adapted through desktop publishing to 
incorporate information spe Jfic to any institution while retaining all its general 
information about teaching and related issues. The development of the handbook 
"template" has already allowed us to serve faculty and SUNYB and Indiana 
Universitv who have their own handbooks. A similar handbook has been 
commissioned by the University of Massachusetts and several other institutions have 
expressed interest in obtaining their own handbooks. 

The project has provided an additional benefit: the rationale for a proposal for 
additional funding. This proposal (to the Sears Roebuck Foundation) is presently 
under consideration and seeks support to establish and staff a center whose specific 
function will be to use the system to assist teaching assistants and new faculty. 

Finally, project staff have presented at several professional meetings and some 
ongoLig, grant-supported research will serve as the focus of a major, sponsored 
symposium to be presented at the annual meeting of the American Educational 
Research Association in March, 1989. 

Although it is difficult to quantify the outcomes of these beginning applications and 
dissemination efforts, all reports from participants or reviewers have been favorable. 
The Northeastern system continues to efficiently handle large numbers of 
evaluations. For example, in the Fall, 1988 quarter, we distributed almost 1500 
course evaluations and we will process and return results this month. This task was 
completed with the equivalent of only four full-time staff. At SUNYB, .he same 
degrees of satisfaction and efficiency have been achieved. 
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B. PURPOSE 



Research and Practice in Evaluating and Improving Teaching 

In the past two decades, student ratings of instruction have been used more and 
more frequently as a measure of teaching performance, course quality, and general 
attitude and/or opinion. At the same time, many methods of providing teaching 
improvemeni scR'ices have been proposed, initiated, and then reduced or 
eliminated because of their cost. Literally thousands of articles and books have been 
written, and hundreds of presentations have been made on the topics of faculty 
evaluation, student ratings, teaching improvement, and the relationships between 
these topics. The results of this work can be simply summarized. 

1. Appropriate methods and instruments for evaluation have been developed 
for evaluating teaching performance in most higher education settings. 

2. Students are valid and reliable raters of the instruction they receive (given 
that the methods and instruments used are appropriate). 

3. Although ratings of specific teacher behaviors may be influenced by context 
(e.g., freshmen might view an instructor's speed of presenting information 
differently than seniors), general ratings (e.g., overall ratings of course or 
instructor) are not consistently or significantly influenced by factors such as 
age, sex, or similar variables. 

4. Evaluation for personnel decision making requires data collected from 
many sources over time. Promotion and tenure decisions should never be 
made on the basis of one kind of information from one source taken from 
one observation or one set of ratings. 

5. Evaluation of teaching performance without associated resources for 
teaching improvement is punitive. 

6. Properly conducted, diagnostic evaluation coupled with resources for 
teaching improvement can lead to better teaching. 



Beyond Resear ch and Practice 

Evaluation is a political issue and faculty development and teaching improvement 
have been expensive services because the teaching consultation model they 
employed. TTiough effective, the process was time-intensive and required a 
substantial number of professional staff even in smaller institutions. Equally 
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important to the success of these programs was the perceived value of the services. 
Research notwithstanding, programs of evaluation and teaching improvement 
require the acceptance of facuhy and administration if they are to succeed. 

The problem facing Northeastern and other institutions of higher education was 
(and is) centered on the question "Can a valid program of evaluation and 
improvement be established?" The question really had two parts: first, in times 
when resources were scarce, could comprehensive teaching evaluation/improvement 
services be provided, and second, would it be possible to successfully implement a 
large-scale program for evaluating and improving teaching? The purpose of the 
FIPSE project was to develop resources which would be added to existing 
mechanisms to create the "system" described above. The development of these 
resources and capabilities affirmatively answered the first question. The second 
question proved much more difficult and, at least at Northeastern University, has 
not yet been answered. The issue of acceptance will be further discussed in the 
"Conclusions" section of this report. 



C. BACKGROUND AND ORIGINS 



Services of The Office of Instructional Development and Evaluation 

At Northeastern University, support for teaching and learning have been centralized 
in a unit presently known as the Office of Instructional Development and Evaluation 
(OIDE). Its mission is to support teaching and learning through designing, 
developing, evaluating and improving instruction. OIDE has provided these services 
to individuals and departments as resources allowed. But Northeastern is a large 
university: the largest private institution in the country. There are over 1,100 full 
and part-time faculty in the day school and another 1,850 in evening programs. 
There are 60 departments and many additional programs oficring instruction. 
Providing comprehensive service to all these clients is not practically possible. 
OIDE had provided evaluation and improvement services to the limit of its 
resources and had used nationally available course evaluation instruments such as 
TABS' from the University of Massachusetts and 'IDEA' from Kansas State 
University to collect information about teaching performance. These instalments 
were useful but three factors argued against their incorporation into a regular 
evaluation/improvement program: 1) the costs associated with external data 
processing or internal development of processing capability; 2) the fact that many 
faculty wanted to be compared to a Northeastern database rather than a national 
norm bank; and 3) the fact that we wanted to combine the best aspects of both 
instruments into the system. 




The Heacher-Course Evaluation Project 

In 1984, OIDE began the Teacher-Course Evaluation Project (TCEP) in an effort to 
address the problem. A model of the evaluation/ improvement system was 
developed and is shown in Figure 1. Essentially, any evaluation and improvement 
model draws on research and theory from three general areas: measurement/ 
evaluation; teaching/learning; and communication/counseling/development (which 
we have labelled "planned change"). Each of these areas informs a geneial function 
of the system. Measurement/evaluation literature informs data collection, handling, 
and analysis. Teaching / learning literature provides the basis for pedagogy and for 
understanding students and the learning process. The literature of psychology, 
communications, organizational development, and other areas provide methods for 
translating reports of i vSults into meaningful, useful information, and for motivating 
faculty to use and benefit from the available resources, and for introducing the 
system into the organization. Another important aspect of the model is that the 
research and practice efforts can be synergistic. Research and theory provide the 
basis for practice, but the use of the system itself can provide data for research which 
can add to the body(ies) of knowledge which inform, the system. 

A TCEP evaluation instrument was developed to assess teaching by focusing on 
actual teacher behaviors. Realizing that resulting data might be used for promotion 
and tenure decisions, some appropriate general questions were added and a means 
of providing comparative reports of results from these items was developed. Finally, 
student, course, and teacher demographic items were added to allow more specific 
analysis of results. The entire process was computer-based for efficiency and speed. 
This process was successfully field tested at Northeastern and the instrument was 
validated. It soon became clear that conducting, analyzing, and reporting the 
evaluations used all available resources and that development of the full "system" 
would require additional support. The original intent for the FIPSE grant was to 
develop the full system with the intent of installing and testing it at Northeastern. 
As the following sections of this report will show, this intent could not survive the 
institutional cutbacks and reorganization that followed. 



The Evaluation Climate at Norttieastern University 

At the time of the FIPSE application, there was no regular or formal university 
procedure for course evaluation. OIDE had been working with a small committee 
representing students, faculty, and administration, to begin a regular evaluation 
service and it was hoped that this effort would lead to a generally accepted process 
based on the best literature and practice as well as the needs of the institution. The 
field tests and validation of the first TCEP questionnaire were being conducted and 
faculty were reporting strong satisfaction with the service and strong confidence in 
its reliability and confidentiality. 
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Unfortunate Politics 

Unfortunately, just at the time when the grant began, the evaluation issue became 
politicized and a two-year debate ensued in the Faculty Senate. As the debate 
proceeded, members of the Student Government Association became more insistent 
in their demands for regular evaluation of all courses. The end result of this activity 
was that the process of developing a comprehensive 'system' was pushed aside by the 
demand for a basic evaluation service which did n t support teaching improvement 
and which used some of the OIDE resources intended for grant work. The impact of 
these events raises issues which are further discussed in the "Evaluation" and 
"Conclusions" sections of this report. 



D. PROJECT DESCRIPTION 



General Strategies for Development 

ITiis was essentially a design/development project intended to acquire resources or 
to produce them from 'scratch'. There were three major types of activity: 1) 
model/theory development for the system and particularly for the computer-based 
teaching improvement consultation process; 2) design/development of 'hands on' 
resources such as print or non-print materials which could be used to help faculty 
improve their instructional or course design skills; and 3) design/development of 
computer resources such as hardware configurations for various environments, 
software, special utilities for data management and administrative tasks, the 
interface between statistical results and the interpretations and/or suggestions 
presented in the narrative report, and documentation. 

A systematic approach to development was employed. Project team memberc were 
individually responsible for developing specific products or carrying out certain 
kinds of activities, as well as for participating in team reviews of processes or 
products. Project tasks were assigned as follows: 



The Project Development Team 

• Project Director: The project director was responsible for project 
administration/coordination/management, product development, and 
internal review/editing. Thus, in addition to budget, scheduling, report 
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writing, coordinating all consultant activities, and other administrative 
functions, the director carried out product development tasks (e.g.: 
executive producer of the "Effective Use of Time in the College 
Classroom" videotape) and edited grant documents (e.g.: the 
monographs on teaching topics). 

• Instructional Design and Systems Coordinator: The coordinator was 
responsible for m.onitoring the instrucnonal design process and 
standardizing the design of the developed products (e.g.: using desktop 
;:ublishing to format printed products for maximum readability, 
organization, appearance, and consistency). A vitally important role of 
the coordinator was the responsibility for designing application software 
and writing specifications used by a systems analyst/programmer hired 
as a long-term development consultant for the grant. The coordinator 
was lead instructional designer for the Northeastern University 
"Handbook for Teachers" (which was a Northeastern contribution to the 
grant) and, using desktop publishing, created the 'template' version of 
the handbook which can be adapted to the needs of any institution. 
Finally, the coordinator acted in a review/editing capacity, 

• Instructional Designer: The instructional designer was responsible for 
developing the teaching topic monographs and for the initial 
development of the annotated database of teaching improvement 
resources in the collection assembled under the grant. The instructional 
designer worked with external consultants who were expert in the 
monograph topics and incorporated their reviews/suggestions into 
revised versions of the monographs. The designer participated in team 
reviews of products and in project planning sessions. 

• Consultants: Expert consultants were used reviewing existing literature 
and/or materials in order to determine those areas in which the grant 
could have greatest impact. These consultants were also used as 
reivewers of developed materials* In the case of the "Effective Use of 
Time in the College Classroom" materials (videotape and workbook), 
the consultants actually developed the workbook, wrote the narrator's 
script for the videotape, and participated in those production and 
post-production activities carried cut at their institution. 



A special use of consultants was made in some research begun with grant support 
(and continuing to be supported by Northeastern at this time). In an effort to 
determine what kinds of system feedback would be most beneficial to faculty and 
administrators, a survey of evaluation knowledge, attitudes, and practice was 
developed for use at several institutions. Twenty-four of the most re spected 
researcher/writer/practitioners in the evaluation and teaching improvement fields 
were asked to respond to the survey and to provide their opinions on its depth, on its 
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correctness with respect to the literature, on its validity, on its usefulness for our 
purposes, and on the potential value of the research to the field. The feedback from 
these experts allowed the development of a more precise questionnaire with items 
validated by the concensus of the experts. Expert feedback also guided the analysis 
of results from the survey and established criteria for measuring respondent 
knowledge in subsequent surveys. Finally, a symposium on research and practice in 
faculty evaluation was successfully proposed for the annual meeting of the American 
Educational Research Association (AERA). In addition to having project staff 
report the research, the symposium will include presentations by five of these same 
experts on topics engendered by the research. 

The most extensive consulting arrangement was with a sysiems analyst/ programmer 
who developed software based on the specifications of the team's systems 
coordinator. This arrangement reflects the team's view that development of 
computer based instruction (or instructionally related materials) is best 
accomplished by using experts in content, design, and computing. Although in this 
case, the content and design expertise were within the project team, a particularly 
high level of computing expertise was required to execute the design. The failure of 
many instructional computing efforts can be traced to the mJstaken assumption that 
only a content expert or programmer or designer is needed for development. 
Without combined expertise (and it 'S almost never found in one person) the 
product will suffer. 

When the grant began, OIDE's staff included two full-time instructional designers 
and one, temporary, clerical assistant, A third instructional design position was 
being advertised. Space was available to house the resources and services which 
would be provided to faculty* The grant was intended to allow the hiring of one 
additional instructional designer and to supplement and review locally developed 
items by using external consultants. Although the grant-supported instructional 
design position wa? filled three months into the grant period, the pressure of the 
situation and limited University resources forced the removal of the unfilled, 
permanent, instructional design position in favor of a full-time, permanent clerical 
position needed to conduct evaluations, This greatly affected OIDE's planned 
strategy for product development under the grant and put a great deal of stress on 
the original tinetable for development and implementation of grant-supported 
products and services. Additionally, the space intended to house the resources and 
services was lost in an administrative reorganization, thus making a field test of the 
service at Northeastern a practical impossibility. Adjusting to these changes 
required extending timetables; using consultants more extensively, and locating an 
external field test site. These strategies were carried out over the grant period. 
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Use of Staff Time 

During the grant period, staff were used as follows (average figures are shown): 

• Project Director = approximately 40% time (contributed) 

• Design/systems Coordiantor = approximately 65% time (contributed) 
with an additional one-fifth, after-hours time (grant-supported). 

• Instructional Designer = 100% time (grant supported). 

• Systems analyst/programmer = the equivalent of one-quarter time 
(grant supported). 

• Development consultants for "Effective Use of Time" = 
approximately 120 hours (grant supported). 

• Other consultants = very few grant hours per person 



Steps in the Development Process 

As previously noted, the project followed a systematic process for development. The 
steps in the process are briefly described in the following outline. 

• Literature review: developing an understanding of the state of the art 
and the standard of practice in general areas relevant to the system 
model as well as the research and theory of evaluation, pedagogy, etc.; if 
necessary, conducting research to guide design and development 
activities. 

• Needs analysis: determining context, resources, requirements, and the 
functions to be served by the system. 

• Audience analysis: identifying system users and their levels of 
knowledge, experience, motivation, etc. 

• Design: using the results of the above analyses; selecting content; 
determining strategies for presentation; incorporating operational, 
ficsal, or other limitations into planning; selecting appropriate media for 
delivering the resource; planning and scheduling events; determining 
evaluative strategies and options for revisions if needed. 

• Development: executing the design for the product. 
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• Evaluation: reviewing and validating the design, concepts, and content 
of products or processes internally or by submitting materials to outside 
experts. 

• Field Testing: using the product and/or process locally or at external 
sites and review of efficiency or effecfVeness by users. 

• Revision: altering products or processes on the basis of review or field 
testing. 

• Dissemination: distributing information about grant-supported 
products, processes, or research; providing grant-supported materials 
to others on a broad scale. 



E. PROJECT RESULTS 



Because the project was delayed, and because practical and institutional limitations 
prevented the installation of a fully developed "system", it is difficult to assess the 
extent to which that system would succeed in an operational setting. However, 
several aspects of the project have been investigated and reported, and many parts 
of the system have been used. The following paragraphs report some of these 
results. 



The Concept 

The concept and theoretical basis for the system have been described at professional 
meetings and in personal communications with the expert consultants employed in 
the grant. (A copy of the system description used at these meetings is attached as 
Appendix *A'.) Reactions have been unanimously favorable, the strongest being 
'This is the best new idea I've seen in five, no, in ten years!" and the weakest being 
"It's a good concept, but the real question is whether you can get faculty to accept it." 
The theoretical basis for the system and the TCEP questionnaire have also been 
' reviewed by experts who concurred that the underlying scholarship was sound. (A 
list of professional meetings at which the system has been described is attached as 
Appendix 'B'. Also included in that Appendix are the names of the expert 
consultants involved in grant-supported development and/or research.) 
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The Teacher-Course Evaluation Project 

The TCEP questionnaire has undergone extensive statistical analysis and has been 
found to be rehable. It assesses constructs and behaviors consistently reported in 
the literature as important facets of instruction. Further, a survey of faculty users of 
the TCEP system revealed high degrees of satisfaction with the service and strong 
confidence in its reliability and con fidentiality. Interim and final reports have oeen 
written for internal dissemination at Northeastern and a comprehensive technical 
report is being written and may be disseminated through ERIC or a similar source. 
Additionally, TCEP results have provided data for evaluation research and our 
intent is to report this research at professional meetings or in appropriate journals. 
(Samples of the TCEP student and instructor q'jesticnnaires are attached as 
Appendix 'C'.) 



The Computing Environment 

The TCEP system has been used to evaluate several thousand courses at 
Northeastern University and has operated in Digital Equipment Corporation's 
VAXA^MS environment on the VAX 11/780, VAX 8650 and the MICRO VAX II. 
We are preparing to implement a microcomputer based system during this academic 
quarter. The equipment configuration will include an "AT' class machine, a laser 
pri.iter, and an optical scanner to read answer sheets. Other equipment associated 
with delivering the service includes a photocopier and a burster/collater to separate 
continuous feed answer sheets after pre-printing. Various other pieces of computing 
hardware are used by OIDE staff and can be 'networked' to the 386 PC, but these 
are not required to operate the system. The field test site at SUNY Buffalo has 
used a PC system for two years, serving about three-hundred faculty. (A brief report 
from SUNY is attached as Appendix 'D'.) A complete desktop publishing system is 
available for production of well-designed, readable documents. 

In both situations, the system has been cost-efficient, accurate, and fast. In fact, the 
only difficulties encountered in the evaluations at Northeastern were caused by 
failures or errors outside the TCEP system (e.g., the university's main computer was 
inoperative; answer sheets were incorrectly printed by computer services; faculty or 
student monitors did not administer evaluations correctly). A strength of the overall 
TCEP process is that these errors were discovered and corrected before any reports 
of results were distributed. The importance of confidentiality 4nd accuracy in any 
system handling sensitive data can not be overemphasized. For the sake of faculty, as 
well as for the institution and the unit delivering the service, it is essential to develop 
means of cross-checking information and exercising quality control. The utilities 
built into the system software produce logs, reports, and various other documents for 
this purpose and these have served us well. 
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The system software and hardware package was designed to be powerful, flexible, 
and portable. That is, many parts of the system can work independently as well as 
within the system. T. j components are designed to allow them to be customized to 
the needs of any institution, and even the full system uses relatively inexpensive 
machinery. Two-year or community colleges could easily afford the evaluation and 
teaching improvement services provided by the system. In fact, the SUNY Buffalo 
service uses an IBM PC XT, a dot matrix printer, and an optical scanner foi reading 
answer sheets. The total cost of such a system is well under $10,000. 



The "Ha^-idbook for Teachers": a Unique and Popular Product 

There has been much positive response to a product contributed to the grant by 
Northeastern. The "Handbook for Teachers" is a document provided to all teaching 
assistants and ne v faculty at Northeastern. It was developed by OIDE and it 
contains general information about teaching, evaluation, and related topics, as v ^11 
as Northeastern-specific information about available re sources, about various local 
procedures, and about campus units providing academically related services. The 
document was published using the desktop publishing system purchased with grant 
funds and a generic version (a 'template*) was produced. The template contains all 
the general information and has space for new, institution-specific information to be 
inserted. At this time, Indiana University and Suny Buffalo are using their own 
versions of the handbook. The University of Massachusetts at Amherst has 
commissioned its own version and several other institutions have expressed interest 
in doing the same. (A copy of the Northeastern 'Handbook' and a brief description 
of the Template" are attached as Appendix 'E'.) 



Computer-based Evaluation and Teaching improvement Bibliographies 

Two, extensive bibliographies have been prepared and these list all the resources 
available at OI^^.. The first contains items on teaching improvement and important 
evaluation iteii ith n^ajor works reviewed and annotated. The purpose of this 
listing is to pro.^de faculty a usLbJe resource of information about teaching. Further, 
because the list is computerized, it can be searched for all relevant titles or authors 
in a given topic. It is this list from which the computer-based consultation system 
will draw the references to be in' cried into the interpretive evaluation reports 
produced by the system. The second lists all the evaluation items at OIDE but is 
more a professional bibliography because of its emphasis on research items. This 
list is also computerized and it, too can be searched for specific topics/authors or for 
more general research purposes. The combined bibliographies include over 2,000 
works in print and other formats, all of which are available at OIDE's offices. These 
bibliographies will be made available to evaluation and teaching improvement 
practitioners and to institutions interested in developing similar sets of resources. 
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(Sample pages from the bibliographies on handbooks, lecturing, feedback/ 
questioning/testing/grading are attached as Appendix T\) 



Other Teaching Improvement Products 

Three teaching improvement products are completed and awaiting final production 
and dissemination. The videotape and workbook entitled "Effective Use of Time in 
the College Classroom", and the two monographs/self-instructional units "Writing a 
Syllabus" and "Selecting Texts and Readings" will soon be prepared for distribution 
at Northeastern, SUNY Buffalo, and selected other institutions (those which 
participated in the evaluation research project). Another monograph, ^'Motivating 
Students" is at the review stage and will be completed as soon as possible. Finally, a 
special monograph on teaching sociology has been written and reviewed ana a fmal 
version will soon be prepared for desktop publishing. (A copy of the videovape is 
included as Appendix but it should be noted that this is a "third generation" copy 
and thus its technical quality is less than the products which will be distributee when 
the materials are reproduced for dissemination.) 



Professional Activities 

Several professional activities have provided opportunities for disseminating 
information as well as for gathering valuable feedback from experts, practitioners, 
faculty, and administrators. Staff associated with the grant have already attended six 
conferences, delivering ten presentations on the system and related evaluation/ 
teaching improvement topics. (See Appendix *B* for a list of meetings.) 

An important aspect of the grant effort has been a research project begun in the 
third year of funding and presently being continued with Northeastern support. The 
research investigates faculty, administrator, and practitioner knowledge, attitudes, 
and practices with respect to the uses of student ratings of instruction. Our intent is 
to determine what kinds of information will be most useful and/or beneficial to 
include when reports of student ratings are prepared. 

For example, we have already determined that it is important to inform faculty about 
the quality of the ratings sample. We have devised a series of statements which 
address specific levels of sample quality based on class size and the percentage of 
those responding to those enrolled. The analysis performed by the ^system* will 
automatically calculate the enrolled and responded numbers and their ratio, and 
then select and print the appropriate message. Thus, if in a class of 36 students only 
13 respond to the evaluation, a "WARNING" would appear in the report of results. 
The warning would note that because fewer that 50% of the class responded, the 
data could not be considered reliable, should be carefuPy reviewed (with qualified 
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assistance) prior to being used as a basis for altering one's teaching, and should 
"NEVER" be presented for use in promotion, tenure, or merit decisions. If the 
sample is adequate and the data can be used without restriction, then a message with 
this information '^"r-ild appear on the report. 

In brief, our research is aimed at identifying as many of these instances of critical 
results as possible, particularly if the research also reveals that faculty and 
administrators might misinterpret or misuse these results. Given this knowledge, we 
can incorporate appropriate safeguards in to the reports produced by the system. 

At the annual meeting of the American Educational Research Association (San 
Francisco: March, 23-30, 1989), the results of our research will be the focus of a 
symposium sponsored by the AERA Faculty Development and Evaluation Special 
Interest Group (SIG). The symposium will include our research report and papers / 
discussion by five of the experts used as consultants in the grant. This national 
exposure will allow wide dissemination of information about the grant and its 
products and will, we hope, generate further interest in the acquisition and use of 
these products by other institutions. 



In evaluating the project, there aie two possible questions to be addressed. The first 
is "Did we accomplish what we proposed, and if so, how successfully?" If the answer 
lO question #1 is "NO", then the alternative question is "What was accomplished and 
what was its importance, its value, and its degree of success with respect to our 
revised goals?" 

The original intent of the FIPSE proposal was to evaluate the individual products 
and processes which were developed, as well as to evaluate the system as whole. 
0: r reasoning was that the individual components (such as the teaching 
improvement monographs) should be usable in their own right as well as being 
useful parts of the system. In effect, the formative aspects of the evaluation of 
system components have been completed through the use of expert consultants, the 
delivery of TCEP evaluation services, and the field test of the limited system at the 
SUNY Buffalo campv " 

However, because research and development necessary to complete the interface 
between ratings data and comprehensive reports were delayed, evaluation of that 
interface has been limited to the consultation and review of experts as described in 
previous sections of this report. Further, as the interface was incomplete, the fiill 
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system could not be implemented and thus, could lot be evaluated except as a 
concept. FinaHy, it was our hope that a small-scale service using the system could be 
initiated at Northeastern, thus providing data on the perceived value and usefulness 
of the system. Surveys of faculty and/or administrative users could have provided 
valuable formative feedback for revisions of the system prior to its use at other 
institutions. 

The results of all our formative evaluations have been positive. FrOin the 
conceptualization cf the system through the teaching improvement products to the 
capabilities built into the software, each component has either received positive 
review or performed well and required only minor revisions. What remains, of 
course, :s to install the system and evaluate its performance, acceptance, and impact 
in the summative sense. 



G. SUMMARY AND CONCLUSIONS 



A Most Difficult issue 

We have concluded that if the products and processes developed under the grant are 
to be useful and valuable, they must exist in a supportive environment. The 
corrolary is that such systems can neither exist nor be introduced into negative 
environments. Further, a system for evaluating and improving teaching can not be 
introduced into even a neutral environment without very careful planning. Without 
the approval and support of faculty and administration, the best services will be 
unused, or worse, misused. 

What constitutes a supportive environment? This is a difficult question to answer 
because different people and different institutions have different definitions of 
support. Yet one of the most critical aspects of this issue is a clear institutional 
definition of mission, direction, expectations, and rewards. If effective teaching is a 
criterion for ^advancement, then the institution must provide not only a proper 
system for evaluating teaching and rewarding effective teaching, but also a means of 
assisting those who fall below criterion. In the absence of such assistance, any 
system for evaluating teaching is simply a weapon to be used against those who dc 
not immediately and consistently reach the criterion. The results are hostility, 
mistrust, unfairness, and the degredation of the system's potential for positive 
impact. 

It is important to remember that the evaluation of faculty performance requires 
much more than the evaluation of teaching. Figure 2 views evaluation as a process 
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of gathering information for specific purposes (its various Voles"). Note that in the 
educational setting, evaluation for promotion, tenure, or merit sits at the top of the 
list of roles. The reason for this is that proper evaluation for merit, promotion, and 
tenure requires more informa tion of more kinds from more sources than does 
evaluation done for other purposes. Thus a second, critical aspect of a supportive 
environment for the system is that student ratings results can never be the sole 
source of information if the role of evaluation is administrative. A complete, public, 
and approved process must be in place. 

An experience at Northeastern demonstrates the importance of having the correct 
environment for evaluation. As noted earlier, just as the TCEP system was being 
readied for broad use, a long and difficult debate began in the Faculty Senate. That 
debate was most certainly influenced by strident demands from the leadership of the 
Student Government Association that campus-wide evaluation be put in place 
immediately. The President announced, prior to the end of the debate, that the 
process desired by the students would be supported and that its results might 
eventually be used for admJnistrative/personnel decision making. The student 
newspaper reported the situation as battle among students, faculty, and 
administration. Even if all other conditions for the acceptance of a properly . 
developed campus-wide evaluation system had been present, these events so 
polarized the situation that its end result was an incomplete system without real 
provision for teaching improvement or the kind of reports of results which could be 
appropriately used in personnel decision making. The comments of faculty clearly 
showed their suspicions that evaluations would be used punitively and that positive 
evaluation results would be meaningless. When such feelings exist, even the 
best-intentioned system will be resisted. 

The FIPSE grant was not intended to support the iniiiatiGn and implementation of a 
large-scale system for evaluating and improving teaching at Northeastern or other 
institutions. We assumed that once it was developed and tested, the system v/ould 
be valued because its merits were apparent: especially in an environment in which 
scholarship, research, and sensitivity to faculty needs were paramount. Although 
this assumption about acceptance was perhaps naive, we feel that the direct goal of 
the grant has been achieved. We have developed "materials for use in a 
comprehensive teaching improvement system**. Our next task is to make that system 
work in a real higher education context. 
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THE-TCEP-SYSTEM 



The Teacher-Course Evaluation Project 



The Teacher-Course Evaluation Project (TCEP) represents an on^going project of 
the Office of Instructional Development and Evaluation at Northeastern University 
to aeate a system which can match student ratings of Instruction with teaching 
improvement resources. To accomplish this, the TCEP evaluation system uses the 
TCEP Questionnaires and TCEP reports of student rating results to piovide 
Instructors with information about self-Instructional materials, lesourocs, and 
services available to Instructors to improve or extend their teaching and course 
design skills. 

TCEP teaching Improvement materials allow Instructors to target specific areas of 
their teaching and course design skills for development. The TCEP system can be 
used with or without the assistance of a teaching Improvement specialist. 

In 1985. the Offioe of In$tnictk>nal Devek^pment and Evaluation received a major 
grant from the Fund to Improve Post-Secondary Education (F.i.P.S.E.) designed 
to further the goals of the TCEP effort. Questtonnaires, evaluation software, 
sel{*instnidionaI mateiiab on teaching fopk:s, and biblk>graph9es of resource 
materials are being devebped under the grant. At the present time, beta test 
versions of TCEP components arc being made available on request on a limited 
basis. 



Major components of the TCEP System include: 



TCEP QUESTIONNAIRES: sets of teacher-course evaluatk>n instruments 



• TCEP Quettionaalm, the prtmaiy ttudent rating form 
(40 Items) 

• TCEP Imtnictoi't Questtonnalre, the parallel form for 
lnstructc;s 

• TCEP Item bank, a collection of additional items for In 
depth assessment of specific aspects or types of Instruction 
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TCEP REPORTS: a coUcction of reports of results ( TCEP system 
software required) including: 



o TCEP iMtnictor's Report: Comprehensive descriptive 
statistics for a course-sections results (mean, standard 
deviation, confidence intervals frequencies, and Instructor's 
scK-ratings) 

# TCEP Instructor's Comparison Reports: Comparison 
statistics showing one course^section results compared to a 
selected group of courses, (means, deviations, T-scorcs, and 
Percentile Group) 

o TCEP AdministiBtlve Report: (I^omparison statistxrs 
showing groups of course-section results sorted by 
department and instructor, compared to a selected group of 
courses, (means, deviations, T-scores, and Percentile Group) 

# TCEP Student Report: frequency statistics only of 5 to 10 
TCEP Items of direct interest to students for the purpose of 
course selection. This form provides a camera-rcady original 
for ti^ pubik:ation of student ratings catalogs. 

9 TCEP Interpretive Report: Comprehensive statistics plus 
individualised comments and listings of available teaching 
improvement resources. 

TCEP GUIDES: pamphlets designed to assist In the interpretation of 
TCEP reports. 

#*'Instnictof's Guide to Interpreting TCEP Results** 

# ^'Admlnlatratoi^s Guld« to Inteq»retins TCEP Results" 



TCEP Teaching Improvement Resource Materials: 

# 'Effective Use of Oass Time* . a video (VHS) with 
workbook for instructors and workshop facilitators 

# ^'Writing, Designing^ and Using the Syllabus"** a print 
module, indudes sample syllabuses 

# ''Selecting Text Materials*, a print module 

# Resource Bibliography Collection, annotated 
bibliographies on teaching improvement and faculty 
development topics 

# Handbook for Teaching Assistants, 60 pages 

# TCEP System Software 

• VAX-VMS based admislrative software, espedal* 
ly suited large scale evaluation opei'atk)ns 

• PC-DOS based administrative software and 
teaching improvement consultation system, espe- 
cially suited to smaller organizations (e.g. com* 
munity colleges, departments, resource centers, 
etc.) 
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Professional meetings attended by project staff: 

First meeting of the National Center for Research to Improve Postsecondary Teaching 
and Learning. Ann Arbor, Michigan: November 1985. Presentation by Project 
Director: "Using student ratings of instruction for research into teaching and 
learning." 

Second National Conference on Instructional Improvement and Evaluation 
Techniques for Colleges and Universities. Tuscon, Arizona: February, 1987. 
Presentation by Project Director: "A cost-efficient system for providing evaluation 
and teaching improvement services." 

Second National Conference on Faculty Evaluation and Development. Orlando, 
Florida: April, 1987. Presentations by Project Director, Instructional 
Design/Systems Coordinator, and Instructional Designer: 'The development of a 
computer-based, diagnostic- prescriptive system for evaluation and teaching 
improvement." 

State University of New York at Buffalo Conference on Teaching Evaluation. Buffalo, 
New York: October, 1987. Presentations by Project Director and Instructional 
Design/Systems Coordinator: "The Teacher-Course Evaluation Project: a tool for 
faculty development." 

Eighth annual meeting of the Society for Teaching and Learning in Higher Education. 
Hamilton, Ontario, Canada: June 1988. Presentation by Project Director and 
Instructional Design/Systems Coordinator: "Reading the ratings. What you don't 
know about student ratings can hurt you. 

First National Conference on the Training and Employment of Teaching Assistants. 
Syracuse, New York: November, 1988. Presentation by grant-supported graduate 
assistant: 'The 'Handbook for Teachers' and the handbook Template'." 



Future presentations: 

Northeastern University "Scholar Days" invited presentation. Boston, Massachusetts: 
March, 1989. Presentation by Project Director and Instructional Design/Systems 
Coordinator: "Using and misusing student ratings data." 

Annual Meeting of the American Educational Research Association. San Francisco, 
California: March, 1989. Presentation/symposium by Project Director and 
Instructional Design/Systems Coordinator: Two different worlds: research and 
practice in the use of student ratings of instruction." 
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Consultants to the FIPSE project: 

(note: before name indicates participant in AERA symposium) 
Dr Philip Abrami, Concordia University 
Dr Lawrence Aleamoni, University of Arizona 
Dr Raoul Arreola, University of Tennessee 
Dr Dab Brandenberg, University of Illinois 
Dr Mary Ann Bunda, Western Michigan University 
Dr William Cashin, Kansas State University 

* Dr John Centra, Syracuse University 

* Dr Peter Cohen, Medical College of Georgia 
Dr Patricia Cranton, Brock University 

* Dr Kenneth Doyle, University of Minnesota 
Dr Glenn Erickson, University of Rhode Island 
Dr Stanford Erickson, no institutional affiliation 
Dr Kenneth Feldman, State Univ. of New York 
Dr Peter Frey, Northwestern University 

* Dr Gerald Gillmore, University of Washington 
Dr George Geis, Ontario Institute for Studies in High Education 
Dr John Keller, University of Florida 
Dr Christopher Knapper, University of Waterloo 
Dr James Kulik, University of Michigan 
Dr Larry Ludlow, Boston College 
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Dr Wilbert McKeachie, University of Michigan 
Dr Phil McKnight, University of Kansas 
Dr Harry Murray, University of Western Ontario 
* Dr Robert Menges, Northwestern University 
Christopher Schmidt, no inrtitutional affiliation 
Dr Thomas Schwen, Indiana University 
Dr Peter Scldin, Pace University 
Dr Gino Sorcinelli, Indiana University 
Dr Mary Deane Sorcinelli, Indiana University 
Dr Maryellen Weimer, Pennsylvania State University 
Dr Luann Wilkerson, Harvard University 
Dr Delivee Wright, University of Nebraska 
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3u The single most important factor determining the grade I expect to 
receive in this course has been: 

A. my ability 

B. my effort 

C the instructor's teaching ability 
D. how difficult (or easy) the course was 
£. the other students in the class 
F. mainly luck 

37. In my program, this course is: 

A. elective in major 
required in major 
C elective but not in majoi 

D. required but not in major 

E. other (e.g., non-credit or audit) 

38. My class is: 

A. freshman 

B. sophomore 

C. middler 

D. junior 
£. senior 

F. graduate student 

39. My overall gradepoint average is: (first quarter freshmen use high school 
overall g.p.a.) 

A. 1.00 ' 1.75 

B. 1,76 ' 2.25 

C. 2.26 ' 2.75 

D. 2.76-3.25 

E. 3.26-4.00 

40. I expect to receive a grade closest to 

A. A 

B. B 

C. C 

D. D 

E. F or U (fail or unsatisfactory) 

F. S (satisfactory, pass) 



YOUR WRITTEN COMf/lENTS ARE ALSO WELCOME! 

Use the BACK of the ANSWER SHEET or a sheet of blank paper to write any 
comments you may have about your instructor, this course, or the rCEP 
questionnaire. 



^•^-"k you for participating in TCEP. 



m - 
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TCEP 
Questionhaire 



Offict of Instnjctionat Otvtloppent vd Evttuation (• division of tht C«nter for Instructional Technotosy) 
Copyright 1985, Northmttm University 



SECTION I: THE INSTRUCTOR 



The following statements describe things effective teachers generally do regardless of 
the subject area or level of students in the course. 

Which of the following options best describes how often each siaiemenl is true? 

A n almost always 
B - more than half of the time 
C - about half of the time 
D - less than hair of the time 
£ « almost never 



OR 



F - this item DOES NOT apply to this course 



The instructor: 



1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 

It. 
12. 

13. 

14. 
15. 

16. 
17. 
18. 

19. 
20. 
21. 
2?.. 



communicates the purposes of each class session and learning activity. 

speaks clearly and audibly when presenting information. 

presents information at a rate 1 can follow. 

indicates which information is essential and which is minor. 

uses examples and illustrations which help clarify the topic being discussed. 

shows important relationships among the copies being treated in this course. 

inspires excitement or interest in the content of this course. 

relates course material to relevant real life situations when poSblL.w. 

asks questions which challenge me to think. 

provides opportunities for me to bring up or discuss issues related to the 
course. 

develops an atmosphere of respect and trust in the classroom. 

manages Qlassroom discussions so that they arc a useful p3rt of my learning 

experience. 

presents activities and materials appropriate for my level of experience 
and ability. 

clears up points of confusion for me. 

provides assistance on individual basis outside of class if I need it. 
gives me regular feedback about how well I am doing in the course, 
states iu advance precisely how my performance is to be evaluated, 
gives tests (ex?ms) that arc a fair and accurate measure of course skills, 
concepts, and information as taught. 

returns exams and assignments quickly enough to benefit mc. 

suggests specific ways I can improve my performance in this course (when needed). 

makf-s effective use of class time. 

is punctual in meeting class and office hour responsibilities. 
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^SECTION II: THE COURSE 

^tCompared to other college courses you have taken: 

tj^^i23. the workload for this course is: 

A. one of the lightest 

B. lighter than average 
C about average 
D. heavier than average 
£. one of the heaviest 

24. the difficulty level of the course activities and materials is: 

A. extremely easy 

B. easier than average 

C. about average 

D. more difficult than average 
£. extremely difficult 

?.5. the textbook(s) and readings used in this course are: 

A. among the best 

B. better than average 

C. about average 

D. worse than average 
£. among the worst 

26. Rate how weM the syllabus, course outline, or other overviews 
provided oy the instructor helped you to understand the goals 
and requirements of this course. 

A. unusually well 

B. better than usual 

C. about as well as usual 

D. worse than usual 

£. not at a*I or no such information was provided. 

27. Rate the usefulness of the outside assignments (writings, reports, 
and special projects) in helping you to Icarn. 

A. extremely useful 

B. more useful than average 

C. of average usefulness 

D. less useful than average 
£. almost useless 

28. Rate how well the various elements of the course (eg., class activities, 
textbooks/readings, and outside assignments) work together in helping 
you icarn. 

A. very well 

B. better than average 

C. about average 

D. worse than average 
£. very poorly 
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29. The course goals or objectives as presented by the instructor were met. 



A. strongly agree 

B. agree more than disagree 

C. agree and disagree, uncertain 

D. disagree more than agree 
£. strongly disagree 

F. no goals or objectives were presented by the instructor 



30. Overall, how much do you feel you have learned in this course? 

A. an exceptional amount 

B. more than usual 

C. about as much as usual 

D. less than usual 

E. almost nothing 

31. What is your overall rating of this instructor's teaching effectiveness 
compared with other college instructors you have had? 

A. one of the most effective 

B. more effective than average 

C. about average 

D. less effective than average 
£. one of the least effective 

32. What is your overall rating of this course? 

A. one of the best 

B. better than average 

C. about average 

D. worse than average 
£. one of the vorst 



SECTION III: THE STUDENT 



Using the scale below, indicate your agreement or disagreement with the 
following statements about yourself. 

A. strongly agree 

B. somewhat agree 

C. mixed feelings, (agree and disagree) 

D. somewhat disagree 
£. strongly disagree 

.F. no opinion or do not uLderstand the question. 

33. My educational background prepared mc with the skills and information 
I need to achieve success in this course. 

34. In my own judgment, what I am being asking to Icarn in this course is 
important. 

35. I picked this course/section based on information about the teaching 
skin of the instructor. ^ , 



36. The primary mode of instruction b this class is 

A« lectQrt(l^ itt8^iictdr)indiexam^ 

B« goap discajiuioa wMi team or collaborativt projects 

C iabor|^ofy,paf^^ 

E txKkp^Mciititu^^ 
F, preteotittk^ 

y^Wim^^ a&d/or exams. 

37. The mats method med for cnrahiadag ttudeiU perf 
this class is assesstneat based oo: 

K tests aad/drenmiooly 

B. papers aiid projects only (no exams) 

C papers aod/orprogects and exams 

D. pen6rmanoei»p^^ 

E nott*print projects (e^coBS^ 

dra\mgs» photographs, videos, films, etc*) 
R studenf s particq|>atioii In dass discussions, group projects, 

ortcamMMrk. 



SECTION IV; Instructional Objectives 



Um the scale bdow to Indicate tiieanphasisyoQ placed in this dass on the 
objcctim described bdow. 

A. « very heavy emphasis 
B« » moderate emphasis 
C « someem{diastt 
D« «* slight emphasis 
E « fioempbasis 

Students: 

33. gaining factual knovdedge (tennindogy, dassificatioos, methods, trends). 

39. learning fundamental pdncq>les, concepts, or theories. 

40. impromg logical thinking, problem*solving, and dcdsion^maldng. 
4L dcvelo|nng spea&c pqfchomotor (manipulative, manuaQ skills. 
41 develo(ring skilk h orgadzint kieas and 

43. opportunities to be oeativc Onui^hative, inventive, oripnal). 

44. developing a favorable attitude toward the subject matter 

45. developing social skills for leadership, teamwork, and group work 



Dcvdoped ondcr a gnat ftom tiM Biod for tiM Irnpm^^ 

(FXP^a) bf Oflkt or Isatmcttoaal Dcvclopmeot sod Bviiu«tk»« 417 Dodfc 
(s div:sioi or tht Oatttr for laMictiooal TeduolofirX 
Noctheaitem Vdhmtdtf, Bottoo, MA 02115 



Instructor's Questionnaire 



SECTION I: THE INSTRUCTOR 



Use the scak below to Ittdknte hoir often statements 1 through 22 arc true 
about YOU as an Instructor for THIS dass. 

A « ahnostahvays 
B « morethanhalfofthetime 
C - about half of the time 
D » letf than half of the time 
E » ahnost never 

F » thb item DOES NOT apply to this course 
As the instroctOTt I: 

1. coiomunicate the purposes of class sessions and leaniing activities. 

2. speak dearly and audiUy when presenting information* 

3. present information at a rate students can follow* 

4. hdicateitditdikformatiottU essential and which is minor 

5. use eiam{^ and ilhistratiom which hdp clarify the topic being discussed 

6. show important relationship among the topics being treated m this course. 
7* in«?ire excitement or itttereit in the content of this course, 

8. relate course materiil to relevant real life situations when posst^ 
9* ask questions irfudidiallenge students to think* 

10. proi^. opportunities for students to bring up or dis 
thecourse. 

11. devek>p an atmoqdiere of req)ect and trust in the classroom* 

12. many dassroom discmdoos so that they are a useful part of students' 
leamifl^ e3^perience» 

13. present acdvitieft and materials appropriate for students' level of 
experience and ability* 

14. dear up points of omfttsko for students. 

15* provide assistance on iacfcidual baris outside of class if students need it 

16. gh«stud(»tsreg:ular feedback about how weU they 

17. state in advance prodselybowstudentperfornunce is to be e^i^^ 

18. give tests (exun^ that are a fair and accurate neasure of course sk^ 
concepts, and inform^^ioa as tatilght 

19* return cams and attjgnmcnti quickly enough to benefit students. 

20. suggest spedfie w^fs students can improve their performance in this 
course (when needed). 

2L make effecth^ use of dau time. 

22, am punctual in meeth^ dau and office hour responsibilities. 
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M l SECTION 11; THE COURSE 



Compared to other college courses (at a similar levels In a eomparable 
content area); 

23. the workload for this course is 

A. one of the lightest 

B. lifter than average 
C about average 

D. heavier than average 
E one of the heaviest 

24. the difTiculty level of the course activities and materials is 

A. extremely easy 

B. easier than average 

C. about average 

D. more difficult than average 

E. extremely difficult 

25. the textbook($) and readings used in this course are 

A. among the best 

B. better than average 

C. about average 

D. worse than average 
E among the worst 

26. Rate how well the s^bus, course outline, or other overviews provided by 
the instructor helped students to understand the goals and rcqtiirements of 
this course. 

A. unusually well 

B. better than usual 

C. about as well as usual 

D. worse than us>ial 

E. not at all or no such information was provided 

27. Rate the usefulness of the outside assignments (writings, reports, and 
special projects) in helping students to learn 

A. extremely useful 

B. more useful than average 
C of average usefiilness 

D. less useful than average 

E. ahnost useless 

28. Rate how weil the various elements of the course (cg^ class activities, 
textbooks/readings, and outside assignments) work together in helping 
students learn. 

A. verywell 

B. better than average 

C. about average 

D. worse than average 

E. very poorly 



29. The course goals or objective as you presented them were met. 

A* ^ongly agree 

B. agree more than disagree 

C agree and disagree, uncertain 

D. disagree more than agree 

E. strongly disagree 

30. Overall, how much do you feel students learned in this course. 

A. an exceptional amount 

B. more than usual 

C about as much as usual 

D. less than usual 

E. ahnost nothing 

31. What is your overall rating of this your teaching effectiveness (in this 
course/section) compared with other college instructors? 

A. one of the most effective 

B. more effective than most 

C. about average 

D. less effective than most 

E. one of the least effective 

32. What ?s your overall rating of this course? 

A. ooeofthebest 

B. better than average 
C about average 

D. worse than avera^te 

E. oneoftheworst 



SECnONIII: OTHER INFORMATION 



Please answer the following Items: 

33. Your rank: (leave blank if not applicable) 

A. fiiU professor (including emeritus) 

B. assosdate professor 

C. assistant professor 

D. instructor 

E. lecturer (including adjunct, senior, and part-time) 

F. teaching assistant 

34. Your years of teaching experience (total at this institution and others) 

A. lessthanone 

B. one to two 

C more than two but less than five 

D. five or more but less than eight 

E. tig^i or more but less than twehf^ 

F. twelve or more 

35. Years of experience, con^t: 

A^ more than twelve but less than twenty 
B. twenty or more 
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UNIVERSITY AT BUFFALO Of«ce of inching Eff^Uveness 

Ellicott Complex 

STATE UNIVERSITY OF NEW YORK Buffalo. New York 14261 



Teacher Coursti Evaluation Project Summary Report 

The State University of New Yorl- at Buffalo has participated in 
the Tearher Course Evaluation PTOject of Northeastern University 
since the fall of 1987. The Office of Teaching Effectiveness has 
been the administrative agency working with the principal 
investigators Jennifer Franklin and Michael Theall, 

For three semesters, SUNY/Buffalo has field-tested the IBM/PC 
version of the stude.it course evaluation program. The first 
semester entailed linking SUNYXBuffalo to the computer system at 
Nor theas tern . The rema i r,i ng two semesters i nvo 1 ved us i ng the pc 
software on our campus. Faculty participants in the field study 
originally came from two sources: faculty volunteers throughout 
the University, faculty teaching Freshmen Seminars? but was 
expanded to includ® first- time faculty in Millard Fillmore 
College (The Evening Division) and faculty in the new World 
Civilization course. Approximately 300 faculty have participated 
in the pilot program. 

Faculty at SUNYXBuffalo are required to use student evaluations 
at the end of each semester. A number of different evaluation 
forms are used, but none of the evaluations offers a formative 
interpretation of data. Using the student evaluati on form in a 
formative way has been helpful to individual faculty in their 
teaching. Perhaps* more important it has raised the consciousness 
of the faculty in general that student evaluation can be a 
diagnostic tool to improve their teaching. 

The spring semester, 1988, required a large block of time to 
famil-iarrze staff with the software and to initiate the program, 
including a number of calls to Northeastern University. Once 
the software was operational, the program ran efficiently and we 
were able to maintain a very short turn around time for faculty. 
This semr * er , using the same version of the software, we have 
been able to run a greater number of courses and still provide 
efficient turn around time. We do know from our experience that 
this program requi res specially hired staff and office staff that 
are able to rearrange their schedules to accommodate the program, 
including use cf computer time. 

The investigators at Northeastern have been able to support all 
facets of our field testing. Consultations by phone have 
answered all of our program and operational concerns. The 
bibliographies about specific areas of teaching have enabled us 
to provide faculty with materials to aid them in making changes 
in their teaching styles. 
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In addition to software and material support for the Teacher 
Course Evaluation Program, SUNYNBuf f a 1 o has utilized the 
compiled teaching ass^istant handbook. With the basic format 
provided through the project we were able to customize the 
materials to apply to our university. The Teaching Assistants. 
Handbook has been pub 1 i shed and over 2000 cop i es distributed to 
teaching assistants and faculty who supervise these persons. 
Our Office continues to get requests for this handbook and has 
received positive feedback from persons familiar with the 
pub 1 i cat ion . 

Being affiliated with the Northeastern Univ6»rsity project* 
Teacher Course Evaluation Program has provide^d services to 
faculty that our rather new operation would not have been able to 
provide. The evaluation service we have offered to faculty has 
raised the consciousness of the faculty to the value of student 
evaluations in improving teaching performance. The support of 
materials designed as part of the PHIPSE program has enabled our 
operation to be more effective in our mission of beirtg a resource 
to faculty on current issues in teaching effectiveness. 
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APPENDIX E 



Please note: only one copy of Appendix E "Handbook for Teachers" is available at 
this time. 
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TEACHING HANDBOOK TEMPLATE 



Developed under a grant to Northeastern University from (he Fund for the Improvement of PostSecondary 
Education, the TEMPLATE is a system for constructing general purpose handl>ooks on teaching topio; 
suitable for r '^aching assistants, associate instructors am^ new faculty. 

The TEMPLATE consists of text which any institution may have modified by the original TEMPLATE 
development team in order create its own unique handbook. The template's core text draws on exemplary 
handbooks from around the country, important literature on post-secondary teaching techniques, and new 
materials written by the development team. The TEMPLATE covers a wide range of topics of particular 
concern to gradi-atc students in teaching roles. New faculty find many of the topics to be a useful introduction 
to their teaching duties. (A cample table of contents is appended to this sheet). 

The TEMPLATE can be economically adapted by its developers to a wide range of postsccondary settings 
through (he medium of desktop publishing. This allows users to have the advantage of already developed and 
comprehensive information on essential tcachmg topics without sacrificing the special local contributio^is 
from within the institution. For institutions with limited resources the TEMPLATE offers a unique 
possibility for creating an instructionally sound handbook without drawing hcavaly on scarce faculty service 
lime or limited financial resources. Moreover, the final handbook product has a clean, typeset appearance. 
The quality of the handbook help^ convey the message that teaching is taken seriously.. 

The goal of Northeastern Universit}' is to disseminate TEMPLATE-bascd handbooks on a non-pro^t basis. 
However, Northeastern University does not distribute copies of the TEMPLATE itself. Instead, the core text 
of the TEMPLATE is customized by the original TEMPLATE development staff to meet each adopting 
mslitution*s local needs. For example, the name of the institution; locaUy written articles; lists oflocal 
resources; names of institutional sponsors, administrators, and local participants in the handbook 
development process; and terms unique to the institution (e.g. "associate instructor" for "teaching assistant") 
can be mcorpatcd into the core text Graphics suppUcd by users (photographs and line drawings) c&a also be 
mcorporatcd into the final product. 

Users provide the TEMPLATE dcvdopcn with any additional text to be inserted, lists of deletioas, and 
necessary information about the modifications. Modifications are made by the developers and a draft version 
IS seat to tLc users for approval Fmal revisions arc made and are foUowed by a final "camera ready" version 
(Mh cover art if requested). Each user institutioa generally makes its own local arrangements for printing. 
The developers can make these trrangeuents if requested to do so. 

RighU to use handbooks coostnicted from the TEMPI^TE are frw 

the oistomizatkm and printing. This cost is mainly determined by the number and scope of the modifications. 
Any institution teccving its own handbook created bom the TEMPLATE is free to reproduce or further 
modify the handbook for use within that institution for as long as needed Text files of the contents of 
handbooks developed from the TEMPLATE are available on 5 1/4 floppy disks for archiving by the adopting 
institution. 

Although the TEMPLATE has only recently become available, users of handbooks developed from the 
TEMPLATE already include Indiana University, SUtc University of New York, Buffalo, and Northeastern 
University. For more information about the TEMPLATE development process, specific terms, and 
limitations for use contact: 



Jennifer Franklin 

Office of Instructional Development and Evaluation, 4k7 Dodge 
Northeastern University 
360 Huntington Avenue 
Boston, MA 02115 
(617) 437-4896 
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APPENDIX F 



TCEP RESOURCE BIBLIOGRAPHY 



Teaching skills: handbooks 



Armstrons, D. G., Denton, J. J., & Savage, T, V., Jr. (1978). Instructional skiiis 

handbook. 302 pgs. Educational Technology Publications, Inc. Englewood 
Cliffs, NJ. 

Provides a 5-component systematic approach to instruction, including specifying 
performance objectives, diagnosing learners, selecting instructional strategies, 
interacting with leamers. and evaluating instruction. 



Christensen, C. R., & Hansen, A. J. (1987). Teaching and the case method: Text, 
cases, and readings. 290 pgs. Harvard Business School Boston. 

A book of cases and readings on teaching with cases, establishing instructor-student 
learning contracts, gaining student respect, leading and controlling classes, retaining 
autnority, leading discussions, etc. 

Eble, K. E. (1979). The crcft of teaching: A guide to mastering the professot^s art. 179 
pgs. Jossey-Bass Publishers San Francisco The Jossey-Bass Series in 
Higher Education. 

Within the framework of a humanistic approach to teaching. Eble provides specific 
advice and realistic suggestions for improving teaching skills, including teaching 
methods, setting classroom atmosphere, handling difficult classroom situations. 



Ericksen, S. C. (1985). The essence of good teaching: Helping students learn and 
remember what they learn. 180 pgs. Jossey-Bass Publishers San 
Francisco. The Jossey-Bass Higher Education Series. 

Examines research on learning, thinking, memory, and motivation to provide insight 
into how instructors can improve teaching methods to enhance student motivation, 
comprehension, retention, and independent thinking. 



Fuhrmann, B. S., & Grasha, A. F. (1983). A practical handbook for college teachers. 
315 pgs* Little, Brown & Company, Boston. 

Discusses skills and ideas necessary for successful teaching, including personal values 
and belief systems, learning and teaching styles, group dynamics, interpersonal 
communication, teaching methods, evaluating students, assessing teaching skills. 
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Teaching skills: handbooks 



2 



Guliette, M. M. (Ed.) (1984). The art and craft of teaching. 130 P3S. Harvard 
University Press, Cambridse. MA. 

A series of essays on college instruction, including the first day of class, lectures, 
questioning techniques, leading discussion sections, teaching essay-writing, grading 
and evaluation. (See separate listings for individual essays.) 



Hyman. R.T. (1974). Ways of teaching. 371 pp. J.B. Lippencott Co. New York. 

A five-part methods book, well documented, with many examples (e.g. dialogue from 
questions). The five parts include: overview, discussion method, recitation and lecture, 
role playing, and question-observing-evaluating. 

Lowman, J. (1984). Mastering the techniques of teaching. 245 pgs. Jossey-Bass 
Publishers. San Francisco. The Jossey-Bass Higher Education Series. 

Examines what constitutes good teaching and shows how to master effective teaching 
techniques, including analyzing and improving classroom performance, selecting and 
organizing dass material, planning course content; evaluating student performance. 



McKeachie, W, J, (1978). Teaching tips: A guidebook for the (beginning coiiege 
teacher. 338 pgs, D. Heath and Company Lexington, MA, 

Adassic introductfiSfTtorcollege teaching - provides tips on all aspects of teaching, from 
course preparation and the first day of dass, to selecting appropriate teaching methods 
and materials, grading and evaluation, and morale, dlsdpline. and order. 



Milton, & Associates (Eds,) (1978), On cotiege teaching: A guide to contemporary 
practices. 404 pgs, Jossey-Bass Publishers San Francisco. The 
Jossey-Bass Series In Social and Benavioral Science and the Jossey-Bass 
Series In Higher Education, 

Acollection of essays (see individual authors for entries) regarding teaching and learning 
issues, induding lecturing, leading discussions, writing objectives, using computers, 
using individualized instruction, developing field experience programs. 



Renner, P, F, (1983), Tlic instructor's survivat kit: A handlHHfkfor teachers of adults. 
Second edldon, 136 pgs. PFR Training Associates, Ltd. Vancouver, BC. 
A how-to book on teaching adults-the fin * nart presents techniques such as setting the 
scene, getting acquainted, involving leaaiers in planning and learning, using 
audio-visual aids; the second part helps develop interpersonal and planning skills. 
Recommended for leaclilng assistants and new faculty. 
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TCEP RESOURCE BIBLIOGRAPHY 



Teaching skills: the lecture method 



Bentley, D. A. (1981). More ammunition for the note-taking feud: The "spaced 

lecture." Improving College and University Teaching. 29(2). pp. 85-87. 

Describes a study the author undertook to determine if the spaced lecture was more 
advantageous (or notetaking than the traditional lecture; found no significant 
differences. 

Bergman, D. (1983, Winter). In defense of lecturing. ADE Bultetin. 76. pp. 49-51. 

Provides a defense of why lecturing is a viable means of teaching. 

Carrier, C. A. (1983). Notetaking research: Implications for the classroom. Journal 
of instructional Development. 6(3). pp. 19-26. 

Presents findings from notetaking literature; states conclusions reached, summarizes 
research supporting it, and implications for college instruction. Also presents 
suggestions on how lecturer behavior can positively affect notetaking practices. 

Cashin, W. E. (1985, September). Improving lectures. /dea Paper No. 14. 4pgs. 
Center for Faculty Evaluation & Development, Kansas State U. 
Manhattan, KS. 

Presents strengths and weaknesses of using the lecture method in teaching; provides a 
set of recommendations for lecturing effectively, including preparation and 
organization, presentation and clarity, stimulation and interest, feedback and 
interaction. 

Dubrow, H., & Wilkinson, J. (1984). The theory and practice of lectures, in M. M. 
Gullette(Ed.) The art and craft of teaching, (pp. 25-37). Harvard 
University Press, MA. 

Presents suggestions on how to improve lecturing skills. 

Fisher, B. F., & Jablonski, C. (1985, February). Questions in the lecture. Teaching 

and Learning €tt Indiana University, pp. 3> 4. University Office for Learning 
Resources, Indiana University, BloomSngton, IN. 

Discusses the three types of questions regulariy heard in lectuves-rhetorical, 
transitional, and discussion questions--and provides suggestions on how to use them 
to their best advantage. 
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Teaching skills: the lecture method 



Frederick, P. J. (1986, Spring). The lively lecture-8 variations. College Teaching. 
34(2). pp. 43 50. 

Discusses criticisms and justifications of the lecture method Piosents 8 ways of livomn^ 
lectures; which to use depends on the goals for a pariicular class penod . tlio skills/noeds 
of *lie students, and professor's personal style. 

Gardner, L. S. (1987). Teaching the huddled masses. Network Newsletter on Co//c^ : 
Teaching. 5(5). pp. 1-3. The Center for Teaching Excellence, Texas A & 
M University. College Station, TX 

Discusses the use of lectures in large classes, and presents Frederick's 8 variations of 
the traditional lecture approach. 



Kemp, J. E., & McBeath, R. J. (1979, July). Preparing lectures (text and workbook). 

53 pgs. Faculty and Instructional Development Office, San Jose State San 
' Jose, CA. 

Self-instructional module on preparing lectures; topics include selecting and organizing 
content, choosing examples and resources, selecting presentation method, creating 
conditions for effective learning, and evaluating lecture effectiveness. 



McKeachie, W. J. (1980). Improving lectures by understanding students' information 
processing. In W. J. McKeachie (Ed.) Learning, cognition, and college 
teaching, (pp. 25-36). Jossey*Bass Publishers San Francisco New 
Directions for Teaching and Learning, No. 2. 

Discusses lectures from the point of view of *'vhat are students trying to do during a 
lecture?' Discusses how students process the information presented in lectures, and how 
lectures can be improved to enhance student leaming. 



Milton, O., & Associates (Eds.) (1978). On college teaching: A guide to cvntemporary 
practices. 404 pgs. Jossey-Bass Publishers, San Francisco. The 
Jossey-Bass Series in Social and Behavioral Science and the Jossey-Bass 
Series In Higher Education. 

A collection of essays (see individual authors for entries) regarding teaching and leaming 
issues, induaing lecturing, leading discussions, writing objectives, using computers, 
using individualized instruction, developing field experience programs. 

Napell, S. M. (1977, April). Updating the lecture. California Journal of Teacher 
Education. 4(1). 

Explores three common misconceptions regardinglecturing, and suggests modifications 
which can help to aeate between teacher and students a sense of the shared adventure 
of learning. 

Napell, S. M. (1979). The lecture. A Teaching Awareness Manual. Piedmont, CA. pgs 

Presents suggestions for presenting good lectures. 
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Teaching skills: the lecture method 



Osterman, D., Christensen, & Coffey* B. (1985, January). The feec back lecture. 



Idea Paper No, 13. 4 pgs. Center for Faculty Evaluation I Development, 
Kansas State U. Manhattan, KS 

Presents (lie idea of feedback lectures, winch overcome many problems mlioront with 
iectunn^ Outlines components of feedback lectures, gives a detailed example, and 
describes expenences of two college instructors using feedback lectures 



Sweeney, J. J., & Reigeluth, C. M. (1984, August). The lecture and instructional 
design: A contradiction in terms? Educational Technology. 24(8) pp. 
7-12. 



Presents advantages and disadvantages of lecturing, and discusses when the lecture 
method is most effective. Provides a general model of a lecture as well as variations 
(e.g., interactive lectures, demonstrations, and interactive demonstrations). 



Thiagarajan, S. (1978, October), Games for unlecturing. Educational Technology 



18(10) pp. 44-45. 

Discusses unlecturing games, which attempt to retain the strengths of the lecture format 
while replacing its passive nature. Gives rationale for unlectunng games, and presents 
instructions on how to play the Press-Conference Game. 



Weaver, R. L., II. (1982), Effective lectudng techniques: Alternatives to classroom 



boredom. New Directions in Teaching. 7, Winter* 31-39* 

On Maryellen Gleason's list of 10 best articles on teaching; discusses the WDA* formula 
for effective lecturing- A (attention). I (interest), D (desire), and A (action). 
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TCEP RESOURCE BIBLIOGRAPHY 



Teaching skills: feedback, questioning, 

testing, and grading 



Feedback: 

Carlson, C. R. (1978). Feedback for learning, pp. 125-152. O. MUton & Associates 
(Eds.) On college teaching: A guide to contemporary practices. 
Jossey-Bass Publishers, San Francisco. The Jossey-Bass Series in Social 
and Behavioral Science and The Jossey-Bass Series In Higher Education. 

Provides a rationale for why rerular feedback to and from students is important to 
teaching and learning; includes suggestions for establishing the proper conditions for 
feedback, and descriptions of several different types of feedback. 

Geis, G. L., Rogers, J., Notar, A., & Pascal, C. E. (1977). Module 4: Characteristics 
of effective Instruction. 118 pgs. The Centre for Learning and 
Development, McGill University Montreal, Quebec. From the series: 
Teaching and Learning: An Individualized Course for Instructors in Higher 
Education. 

Presents 10 characteristics of effective instruction: tests prerequisite skills, gets feedback, 
adapts to individual differences, is flexible, allows active student learning, gives 
feedback, motivates students, cumulative review, testing, previews topics. 



Questioning: 

Fisher, B. F., & Jablonski, C. (1985, February). Questions in the lecture. Teaching 
and Lecamlng at Indiana University, pp. 3-4. University Office for Learning 
Resources, Indiana University Bloomington, IN. 

Discusses the three types of questions regularly heard In lectures-rhetorical, 
transitional, and discussion questions-and provides suggestions on how to use them 
to their test advantage. 

Hyman, R.T. (1974). Ways of teaching. 371 pgs. J.B. Lippencott Co. New York. 

A five-part methods book, well documented, with many examples including dialogue 
from questions. The parts include: overview; discussion method; recitation and lecture; 
role playing; and qucstioning-answerlng-evaluating. 
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Teaching skills: feedback, questioning, testing, grading 



Napell, S. M. (1978, Winter). Using questions to enhance classroom learning. 
Education. 99(2). 

Discusses how questions can subvert or enhance teaching goals; classifies and analyzes 
classroom questions, discusses components of effective questioning strategies, and 
offers methods for using questions to stimulate thinking and enhance learning 



Sanders, N. M. (1966). Classroom questions: What kinds? 176 pgs. Harper & Row, 
Nevi ^ork. 

Presents a taxonomy of questions (memory, translation, interpretation, application, 
analysis, synthesis, evaluation) to correspond with Bloom's taxonomy of learning 
outcomes; discusses each type of question and how to use questions effectively. 



Testing and Grading: 

Cashin, W. E. (1987, January). Improving essay tests. JdeoPoper No. 17. 4 pgr. 
Center for Faculty Evaluation & Development, Kansas State U. 
Manhattan, KS. 

Discusses strengths and weaknesses of essay tests; provides reconnmendations on when 
essay questions should be used, constructing essay tests, and scoring essay tests. 



Frisbie, D. A., Diamond, N. A., & Ory, J. C. (1983), Assigning course grades. 30 pgs. 
Office of Instructional Resources, University of Illinois, 
Urbana*Champaign, IL. The Assessment and Improvement of Instruction. 

Meant as a reference for instructors wishing to establish grading policies and practices, 
or to review and revise current practices. Provides guidelines on assigning grad<=»s, 
grading multi-sectioned courses, evaluating grading policies. 



Jedrey, C. M. (1984). Grading and evaluation, pp. 103-115. M. M. Gullette (Ed.) The 
tat and crcft of teaching. Harvcid University Press. Cambridge, MA. 

Discusses the importance of considering how and why students are being graded; 
discusses issues to be considered when deciding how to evaluate student performance. 



MacDonald, J. (1986, September). But, how do you know if a s good test ? Part II: 
EssAys. EKU * Te€»ching. 3(1)« pgs. 3*4. Eactem Kentucky University, 
Richmond, KY. 

Essay questions test writing and organizational skills, and creative application of material 
leamec, however, they are difficult to score reliablly. Article ^ves hints on writing good 
essay questions, and how to achieve usable, reasonably valid scores. 
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APPENDIX G 



Please note: only one copy of Appendix G "Effective Use of Time in the College 
Classroom" is available at this time. 
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APPENDIX X 'SPECIAL COMMENTS FOR FIPSE 
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Special comments for FIPSE 



Questio #1. What forms of assistance were helpful? How can 'ne Fund 
more ef actively work with projects? 

In this project, FIPSE staff were very helpful in suggesting ways in which I could 
overcome obstacles to achieving project goals. These obstacles were most often 
related to institutional constraints or unforseen practical limitations on the extent to 
which anticipated university support could be delivered. Essentially, my questions to 
FIPSE focused on how to take such problems into account and how to alter the 
course of the project without significantly or drastically altering its goals. FIPSE 
staff explained the Fund's position on changes and how I could incorporate such 
changes into reapplications and other documents. These conversations and 
explanations relieved a good deal of anxiety felt by project staff. Eventually, with the 
help and guidance of FIPSE staff, we were able to achieve the major objectives of 
the project. 

When this grant began, I was unaware of the FIPSE Technology Study Group. My 
first FIPSE contact made me aware of FTSG and I became a member. Thiy vvas a 
profitable and interesting side benefit of being a project director and I might not 
have been aware of it if my FIPSE contact had not involved me immediately. 

^jattended one director's meeting, was unable to attend one, and through some 
embarrassing misreading of my travel schedule niissed'th'€rthTrd:--rfoun{Lthe,,one 
meeting I^rd attend to be useful in several'wa>/sr Asanew project director, I was 
helped considerably by the orientation/information meetings designed to help folks 
like myself get started. I also enjoyed and profited from meeting other project 
directors and sharing information with them. The general presentations were 
interesting and the following dialogues equally valuable. 

Beyond these kinds of interactions, I didn't seek much assistance frpm^^j^ 
intera^tionjjd^ ,This was due to limited time and the fact that our 

concentration was on achieving those goals we had set (and reset) as time 
progressed. With more time, I mightjiave been more in touch and had a better 
opportunity to^become more involved with FIPSE or with other projects. 

What can FIPSE do? This is difficult to answer because I think each project has its 
own direction and needs. It's possible that the agendas of the Fund and its staff 
might match the goals of one project very well and be less attuned to the needs of 
another project. In this sense, I think that more regular, FIPSE initiated contact 
with project directors might be useful. I had opportunity to interact at length only 
wilfif my first contact. But, at the time, FIPSE was better funded an3 had more staff. 
Trealize the limitations of the present are similar to those I have faced here and 
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thus, can not make the case that I should have had more chances to interact length 
with my other contacts. 



Question #2. What should the Fund consider for future grants in my area 
of interest. 

My personal experience has been borne out by the rather broad scale research we 
have carried out ... and the conclusions I have reached are somewhat pessimistic. I 
believe that the critical element for any valid instructional or faculty evaluation/ 
development program is the extent to which it is supported. This support must come 
from two quarters: administration and faculty. Let me elaborate. 

As the report noted, faculty must trust, believe in, and accept any program offering 
evaluation/development services or products. But in the area of evaluation, faculty 
have a habit of disregarding research they don't like. Though there are many valid 
reasons that faculty could be suspicious of imposed evaluation programs, their 
arguments against any kind of evaluation are surpricingly uninformed. They prefer 
to accept anecdotal evidence to research: "I know a guy who gave all 'A's and 
increased all his ratings". They exhibit the kind of scholarship for which they would 
fail their students: "I don't care about the research. I just know!" And they cling to 
ideas that have no basis in the research: Tough teachers always get low ratings." 
They often believe that anyone who is not a faculty member is not credible and has 
nothing of value to say: "...really rotten teachers ... should be forced to shape up or 
leave. But that is a job for strong deans and chairmen, not student "evaluators" and 
educationists like yourseif." (A real quote from a Northeastern faculty member in 
response to the evaluation survey distributed at NU .. thank goodness this person is 
one of a very small, but usually present and very vocal minority) 

The conclusion is that some evidence of faculty acceptance and support for the 
project should be present. Though I do not feel that a faculty senate or similar body 
should have veto authority over the research or grant activities of non-faculty staff or 
units, it seems to me that apriori faculty commitment to the proposal and to support 
of its intended services (for, after all, FIPSE grants in this area are usually intended " 
to design, develop, or initiate services) will help to insure the. continuity of these 
services. If faculty are not directiyaaYolvedinthe writing the proposal or carrying 
out the project, then some measure-of-their-confidence is important in considering 
the proposal's value. 

The other unfortunate aspect of jhe^ation-is-that institutions (and/or:,theii- 
.^Jeaders) a'-e quite willing to reduce or remove support for development programs 
when resources are tight. While there is no money to pay for evaluation or 
development staff, for example, several hundred-thousand dollars are spent on 
landscaping and purely decorative projects. Our athletic coffers are amply supplied 
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with travel funds, but there is no budget for books or journals focusing on 
teaching/learning/evaluation issues. 

These observations lead me to suggest that in development and/or evaluation grants, 
FIPSE should insist on more clearly defined institutional contributions and 
cOmmnm^ when grants are giveg^gjnd should check to be sujfi that these 
commitments are honored during the^gfianUln the case of this grant, for example, if 
we had cFearly defined our intent to use certain spaces to house a faculty teaching 
resource center, had received approval to submit the grant, and FIPSE had 
specified in its docun?ents that the Fund would support certain activities for certain 
purposes with certain institutional commitments, then the spaces in question might 
not have been lost (as indeed they were) when internal administrative changes were 
made. Another important aspect of this issue is the need for continued institutional 
support after the grant. In our case, the existing office and its staff would have 
remained anyway, but beyond this, there would have been committed staff, space, 
and resources to enlarge, or at the very least, c ^tinue to refine and improve the 
products and/or services which the grant had anowed us to develop. M a side note, 
although a FIPSE dissemination grant \yould have been most useful for us, I decided 
not to apply because I was unsure of the immediate or long term future at 
Northeastern. Our administration was changing and there was no way to assess 
future levels of support. This coupled with the public debate about evaluation 
described in the report, led me to the conclusion that it v;ould be unwise to apply. 
As the situation has stabilized, I now wish I had applied, The grant would have 
allowed us to get the system and its components out into the public view more 
quickly and efficiently. 

I do not mean these comments to suggest that our grant commitment was denied, 
that we did not accomplish our basic goals, or that this okice and its functions will be 
reduced or eliminated. This would be both incorrect and foolish. Rajher, my point 
is that had we been foiced to be more specific about goals and commitm^ents, then 
the institution would have had to make more specific apriori decisions. If 
Northeastern had said "Yes, we want this space to become a teaching resource 
center for faculty" then we could have moved in that direction. If Northeastern had 
said "No, we can not presently commit this space for ^tfiis use nor can we promise that 
it will be available at the end of the grant Deriod" then our proposal could have been 
altered. The end result would have been a more realistic proposal and a better 
match of promise and accomplishment. 

This does not necessarily mean that only those institutions with existing programs, 
substantial endowments, or long *track records' should be funded because they can 
make such commitments. In many cases, it is the smaller institution which truly 
needs the assistance in starting such programs. The emphasis should be on 
follow-through. Yet even this approach has its drawbacks because smaller 
institutions or those programs dealing with non-traditional learners might be best 
served by adapting and adopting rather than by Ve-inventing wheels'. In the case of 
evaluation/development grants, this implies a clear review of these alternatives 



before funding, so that needs can be clearly identified. It also imb ues that the 
institution has on staff, people qualified to perform both the apriori and committed 
tasks. Generally, this means instructional design/evaluation staff. 

How can this be accomplished? I have two suggestions. First^ it might be 
reasonaWe tQ.pxaYide^exploratoiy-gcajDjt^.^0f,6n^^ to 1 year duration and to 
tighten the re-application process so that the above factors can be weighed. Second, 
provision of funds in this area might include the requiremenYTERars'dftie" Of the grant 
money be spent of expert consultation in evaluation/development during the 
exploratory period. The list of consultants in the report (with my name and Jennifer 
Franklin's added) might be a starting place to which FIPSE could direct its 
applicants. Our experience at the field test site at SUNY Buffalo (where we were 
brought in as consultants to talk to the faculty senate and to administrators) showed 
the value of this approach. We believe that having outsiders involved reduces the 
concern that the institution is preparing for a mandatory process or 'trying to sneak 
something under the table'. At SUNY, the decision about going ahead with the field 
test of the TCEP system was not made until these representative groups expressed 
approval. 



Summary comments 

The grant experience has been profitable and I believe that we have made, and will 
be able to continue to make contributions to both the research and practice in the 
field. I also believe that we will be able to disseminate the system and its products 
though this may not happen^in the immediate future. Given our experiences at 
Northeastern over the past three years, it is safe to sry that without the FIPSE grant, 
we would have been unable to develop the system or its components. 
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